Passenger detection device, passenger detection system, and passenger detection method

ABSTRACT

A passenger detection device (10) includes a face image obtaining unit (12) obtaining a face image captured in a state in which a passenger in a vehicle irradiated with irradiation light from light sources. The device further includes: a reflected-image position detecting unit (14) detecting, from the face image, positions of respective reflected images of the light sources reflected on eyeglasses worn by the passenger; an eyeglasses feature detecting unit (15) detecting, from the face image, positions of center points of lenses as feature information of the eyeglasses; a head position detecting unit (16) detecting a head position of the passenger on the basis of the positions of respective reflected images; and a face orientation detecting unit (17) detecting face orientation of the passenger on the basis of relation between the head position of the passenger, the positions of respective reflected images, and the positions of center points of lenses.

TECHNICAL FIELD

The invention relates to a passenger detection device, a passengerdetection system, and a passenger detection method for detecting faceorientation of a passenger in a vehicle.

BACKGROUND ART

A conventional driver sensing device that detects the face orientationof a passenger has a problem that, when the passenger wears eyeglassesor sunglasses, feature points of the face cannot be detected and thusthe face orientation cannot be detected. Thus, in Patent Literature 1, aface state detecting device is proposed that detects the faceorientation on the basis of images of vehicle body structures reflectedon the lenses of the eyeglasses, etc., worn by a passenger and thepositions of the images.

CITATION LIST Patent Literatures

Patent Literature 1: JP 2004-70514 A

SUMMARY OF INVENTION Technical Problem

The face state detecting device according to Patent Literature 1 isconfigured in the above-described manner and thus has a problem that,when reflected images do not appear on the lenses of eyeglasses, etc.,at night with no outside light, in a cloudy day, or the like, the faceorientation cannot be detected. In addition, there is another problemthat since extraction of images of vehicle body structures reflected onthe lenses is performed by image processing, a high-quality camera isrequired, resulting in high cost.

The present invention is made to solve the above problems, and an objectof the invention is to detect the face orientation of a passenger alwaysat low cost without being affected by the time, weather, and the like.

Solution to Problem

A passenger detection device according to the invention includes: a faceimage obtaining unit obtaining a face image of a passenger in a vehicle,the face image being captured in a state in which the passenger isirradiated with irradiation light from a plurality of light sources; areflected-image position detecting unit detecting positions ofrespective reflected images of the plurality of light sources from theface image obtained by the face image obtaining unit, the reflectedimages being reflected on eyeglasses worn by the passenger; aneyeglasses feature detecting unit detecting, from the face imageobtained by the face image obtaining unit, feature informationindicating a position or shape of the eyeglasses which corresponds toface orientation of the passenger; a head position detecting unitdetecting a head position of the passenger on a basis of the positionsof respective reflected images detected by the reflected-image positiondetecting unit; and a face orientation detecting unit detecting faceorientation of the passenger on a basis of the head position of thepassenger detected by the head position detecting unit, the positions ofrespective reflected images detected by the reflected-image positiondetecting unit, and the feature information of the eyeglasses detectedby the eyeglasses feature detecting unit.

Advantageous Effects of Invention

According to the invention, the face orientation of a passenger in avehicle is detected using a face image captured in a state in which thepassenger is irradiated with an irradiation light from a plurality oflight sources, and thus, the face orientation of the passenger canalways be detected at low cost without being affected by the time,weather, and the like.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an exemplary configuration of apassenger detection system according to a first embodiment of theinvention;

FIG. 2A is a diagram exemplifying a positional relation amongirradiating units, an imaging unit, and a passenger of the firstembodiment, and FIGS. 2B to 2E are diagrams showing exemplaryarrangements of a plurality of light sources included in an irradiatingunit;

FIG. 3 is a flowchart showing an operation of a passenger detectiondevice according to the first embodiment;

FIGS. 4A to 4C are diagrams for explaining a method of detecting a headposition performed by the passenger detection device according to thefirst embodiment;

FIGS. 5A to 5F are diagrams for explaining a positional relation betweena lens of eyeglasses and reflected images appeared on the lens;

FIG. 6 is a diagram showing the positions of reflected images reflectedon eyeglasses lenses when a passenger wearing eyeglasses orients his/herface in various directions;

FIG. 7 is a diagram for explaining a method of detecting the horizontalangle and vertical angle of the face orientation performed by thepassenger detection device according to the first embodiment;

FIGS. 8A and 8B are diagrams for explaining a method of detecting aninclination angle of the face orientation performed by the passengerdetection device according to the first embodiment;

FIG. 9 is a block diagram showing an exemplary configuration of apassenger detection system according to a second embodiment of theinvention;

FIG. 10 is a flowchart showing an operation of a passenger detectiondevice according to the second embodiment;

FIG. 11 is a diagram for explaining the shapes of eyeglasses lenses ofthe second embodiment;

FIGS. 12A, 12B, and 12C are diagrams for explaining a method ofdetecting the face orientation performed by the passenger detectiondevice according to the second embodiment;

FIG. 13 is a block diagram showing an exemplary configuration of apassenger detection system according to a third embodiment of theinvention;

FIG. 14 is a flowchart showing an operation of a passenger detectiondevice according to the third embodiment;

FIG. 15 is a block diagram showing an exemplary configuration of apassenger detection system according to a fourth embodiment of theinvention;

FIG. 16 is a flowchart showing an operation of a passenger detectiondevice according to the fourth embodiment;

FIG. 17 shows the continuation of the flowchart shown in FIG. 16; and

FIG. 18 is a hardware configuration diagram of the passenger detectionsystem according to each embodiment of the invention.

DESCRIPTION OF EMBODIMENTS

To describe the invention in more detail, some embodiments of thepresent invention will be described below with reference to theaccompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing an exemplary configuration of apassenger detection system according to a first embodiment of theinvention. The passenger detection system shown in FIG. 1 includes anirradiating unit 1, an imaging unit 2, and a passenger detection device10. In each of the following embodiments of the invention, an example ofa passenger detection system mounted on an automobile is explained.Alternatively, the passenger detection system can be mounted on avehicle such as a railway vehicle, a ship, an aircraft, or the like.

The irradiating unit 1 includes a plurality of light sources. Theirradiating unit 1 is installed in a vehicle interior in front of apassenger, and irradiates the passenger with irradiation light from theplurality of light sources.

The imaging unit 2 includes one or a plurality of cameras. The imagingunit 2 is installed in the vehicle interior, and captures an image of apassenger's face and outputs the image as a face image.

For example, light emitting diodes (LEDs) that irradiate visible lightmay be used as a light sources of the irradiating unit 1, and a camerahaving sensitivity in a range of visible light may be used as theimaging unit 2. Alternatively, LEDs that irradiate invisible light suchas infrared rays may be used as the light sources of the irradiatingunit 1, and a camera having sensitivity in a range of the invisiblelight may be used as the imaging unit 2. When invisible light LEDs areused, even if the passenger is subjected to irradiation at night, thepassenger is not dazzled.

FIG. 2A shows an example of a positional relation among the irradiatingunits 1, the imaging unit 2, and a passenger. In the example of FIG. 2A,the irradiating units 1 and the imaging unit 2 are installed in front ofa passenger. The two irradiating units 1, each including five lightsources, are arranged on the left and right sides of the imaging unit 2.

In the example of FIG. 2A, visible light LEDs are used, and the twoirradiating units 1 are installed on both sides of the imaging unit 2 sothat a passenger's face is not shadowed. When infrared ray LEDs areused, only one irradiating unit 1 may be used.

Further, as shown in FIG. 2A, a single casing into which the irradiatingunits 1 and the imaging unit 2 are integrated may be formed, or a casingfor the irradiating units 1 and a casing for the imaging unit 2 may beseparated.

In the following, as shown in FIG. 2A, the horizontal direction isrepresented as the x-axis direction, the vertical direction isrepresented as the y-axis direction, and the depth direction isrepresented as the z-axis direction.

FIGS. 2B, 2C, 2D, and 2E are diagrams showing exemplary arrangements ofa plurality of light sources included in an irradiating unit 1. Anirradiating unit 1 shown in FIG. 2B is configured such that four lightsources are arranged on up, down, left, and right side. An irradiatingunit 1 shown in FIG. 2C is configured such that five light sources arearranged to form a cross shape. An irradiating unit 1 shown in FIG. 2Dis configured such that five light sources are arranged to form an Xshape. An irradiating unit 1 shown in FIG. 2E is configured such thateight light sources are arranged to form a cross shape.

As shown in the drawings, it is preferable for the irradiating unit 1 toform a characteristic arrangement such as a vertical line, a horizontalline, an oblique line, or a cross shape formed by a combination thereof,or a triangular shape, that allows to distinguish the irradiation lightfrom the irradiating unit 1 from light other than the irradiation lightupon reflection on eyeglasses lenses. For reference, light other thanthe irradiation light from the irradiating unit 1 includes, for example,external light from the outside of the vehicle, and external lightreflected by a lens has a planar shape or a single point shape.

The passenger detection device 10 includes a control unit 11, a faceimage obtaining unit 12, a face detecting unit 13, a reflected-imageposition detecting unit 14, an eyeglasses feature detecting unit 15, ahead position detecting unit 16, and a face orientation detecting unit17. The passenger detection device 10 detects the face orientation of apassenger wearing eyeglasses in general, including eyeglasses for visioncorrection, sunglasses or goggles for eye protection, or the like.

The control unit 11 controls the operation of the irradiating unit 1,the imaging unit 2, and the passenger detection device 10. Specifically,the control unit 11 instructs the irradiating unit 1 to control theturning on and off of the plurality of light sources. Further, thecontrol unit 11 instructs the imaging unit 2 to capture an image of thepassenger being irradiated with irradiation light from the irradiatingunit 1 with the camera. Moreover, the control unit 11 instructs the faceimage obtaining unit 12, the face detecting unit 13, the reflected-imageposition detecting unit 14, the eyeglasses feature detecting unit 15,the head position detecting unit 16, and the face orientation detectingunit 17 in the passenger detection device 10 to control the timing ofoperation or to control the receiving and outputting of information.

The face image obtaining unit 12 obtains a face image from the imagingunit 2 and outputs the face image to the control unit 11.

The face detecting unit 13 receives the face image from the imaging unit2 through the control unit 11, detects a face from the face image, andoutputs the face detection result to the control unit 11. The facedetecting unit 13 is, for example, a classification unit using commonalgorithms such as Adaboost and Cascade. The passenger detection device10 may not be configured to include the face detecting unit 13, and anexternal device such as the imaging unit 2 may be configured to includethe face detecting unit 13. When an external device is configured toinclude the face detecting unit 13, the face image obtaining unit 12obtains a face image and a face detection result from the face detectingunit 13 in the external device and outputs the face image and the facedetection result to the control unit 11.

The reflected-image position detecting unit 14 receives the face imageand the face detection result through the control unit 11 and detectsreflected images which have the same shape as the arrangement of theirradiating unit 1 from within a face detection frame in the face image,the reflected images being reflected on lenses of eyeglasses worn by thepassenger. For example, when the plurality of light sources of theirradiating unit 1 are arranged as shown in FIG. 2C, the reflectedimages reflected on the lenses of the eyeglasses in the face image alsobecome a plurality of white dots such as those of FIG. 2C. It is assumedthat information about the arrangement of the irradiating unit 1 todetect reflected images is given to the reflected-image positiondetecting unit 14 in advance. For detecting the reflected images, thereflected-image position detecting unit 14 may perform edge detectionwithin the face detection frame or may perform a comparison amongluminance values.

When the reflected-image position detecting unit 14 detects reflectedimages which have the same shape as the arrangement of the irradiatingunit 1, the reflected-image position detecting unit 14 determines thatthe passenger wears eyeglasses, and detects the positions of therespective reflected images corresponding to the plurality of lightsources of the irradiating unit 1. Then, the reflected-image positiondetecting unit 14 outputs the detection results of the positions of therespective reflected images to the control unit 11.

On the other hand, when the reflected-image position detecting unit 14does not detects reflected images which have the same shape as thearrangement of the irradiating unit 1, the reflected-image positiondetecting unit 14 determines that the passenger does not weareyeglasses, and outputs the determination result to the control unit 11.

The eyeglasses feature detecting unit 15 receives, through the controlunit 11, the face image and the face detection result and detects lensesof the eyeglasses worn by the passenger or frames fixing the lenses fromwithin the face detection frame in the face image. Then, the eyeglassesfeature detecting unit 15 detects, using detection results of the lensesor frames, positions of the center points of lens shapes as featureinformation indicating the position of the eyeglasses corresponding tothe face orientation of the passenger, and outputs the detection resultsto the control unit 11. To detect lenses or frames, the eyeglassesfeature detecting unit 15 may perform edge detection within the facedetection frame or may perform a comparison among luminance values.

Note that the reflected-image position detecting unit 14 and theeyeglasses feature detecting unit 15, for example, detect the positionsof the respective reflected images and the positions of the centerpoints of the lenses with reference to a coordinate system with x- andy-axes set in the face image.

The head position detecting unit 16 receives, through the control unit11, the detection results of the positions of the respective reflectedimages, and determines distances between the reflected images from thepositions of the respective reflected images and thereby detects a headposition of the passenger. The head position detecting unit 16 outputsthe detection result of the head position to the control unit 11. Amethod of detecting a head position will be described in detail later.

The face orientation detecting unit 17 detects the face orientation ofthe passenger, and has a function of detecting the face orientation fora case of a passenger wearing eyeglasses and a function of detecting theface orientation for a case of a passenger not wearing eyeglasses.However, the function of detecting face orientation for a case of thepassenger not wearing eyeglasses may not be included.

When the passenger wears eyeglasses, the face orientation detecting unit17 receives, through the control unit 11, the detection results of thepositions of the respective reflected images, the positions of thecenter points of the lenses, and the head position, and detects the faceorientation of the passenger on the basis of a positional relation amongthe reflected images, the center points of the lenses, and the headposition. The face orientation detecting unit 17 outputs the detectionresult of the face orientation to the control unit 11. A method ofdetecting face orientation will be described in detail later.

When the passenger does not wear eyeglasses, the face orientationdetecting unit 17 receives, through the control unit 11, the face imageand the face detection result, detects the face orientation of thepassenger, and outputs the detection result to the control unit 11. Inthis case, the face orientation detecting unit 17 detects the faceorientation using a known technique and may, for example, performdetection of the face orientation by detecting feature points of theface corresponding to the eyes, nose, or mouth from within the facedetection frame, or perform detection of the face orientation by imageprocessing.

Next, with reference to the flowchart of FIG. 3, an operation of thepassenger detection device 10 according to the present embodiment willbe described. The passenger detection device 10 repeatedly performsprocesses in the flowchart shown in FIG. 3. In the followingdescription, it is assumed that the irradiating unit 1 has five lightsources arranged as shown in FIG. 2C.

At step ST10, the face image obtaining unit 12 obtains a face imagecaptured by the imaging unit 2. The control unit 11 outputs the faceimage obtained by the face image obtaining unit 12 to the face detectingunit 13. The face detecting unit 13 detects a passenger's face from theface image received from the control unit 11, and outputs the facedetection result to the control unit 11.

At step ST11, the reflected-image position detecting unit 14 receivesthe face image and the face detection result from the control unit 11and detects reflected images which have the same shape as thearrangement of the irradiating unit 1 from within the face detectionframe. If the reflected-image position detecting unit 14 detectsreflected images which have the same shape as the arrangement of theirradiating unit 1 (“YES” at step ST11), the reflected-image positiondetecting unit 14 determines that the passenger wears eyeglasses, andproceeds to step ST12.

On the other hand, if the reflected-image position detecting unit 14does not detect reflected images which have the same shape as thearrangement of the irradiating unit 1 (“NO” at step ST11), thereflected-image position detecting unit 14 determines that the passengerdoes not wear eyeglasses, and proceeds to step ST18.

At step ST12, the reflected-image position detecting unit 14 detectspositions of the respective reflected images and outputs the detectionresults to the control unit 11.

At step ST13, the head position detecting unit 16 receives the detectionresults of the positions of the respective reflected images from thecontrol unit 11, detects a head position of the passenger, and outputsthe detection result to the control unit 11.

The coordinate value of the position of the head position in thehorizontal direction (x-axis direction) and that in the verticaldirection (y-axis direction) are calculated using the coordinate values(x, y) of the positions of the reflected images in the face image.

Further, the coordinate value of the position of the head in the depthdirection (z-axis direction) is calculated using distances among thereflected images.

Now, a method for calculating the coordinate value of the position ofthe head in the depth direction will be described.

FIGS. 4A, 4B, and 4C are diagrams for explaining a method of detecting ahead position, and show five reflected images reflected on an eyeglasseslens. For example, it is assumed that, when the distance between thepassenger and the irradiating unit 1 is 80 cm, the distances amongreflected images are three pixels, as shown in FIG. 4A. When thepassenger moves forward and the passenger and the irradiating unit 1 getcloser to each other to the distance of 70 cm, the distances amongreflected images are, as shown in FIG. 4B, one pixel, namely, the spacesamong the reflected images become narrower. On the other hand, when thepassenger moves backward and the passenger and the irradiating unit 1get away from each other to the distance of 90 cm, the distances amongreflected images are, as shown in FIG. 4C, five pixels, namely, thespaces among the reflected images become wider. The head positiondetecting unit 16 calculates the coordinate value of the position of thehead in the depth direction on the basis of the ratios between theactual distances among the plurality of light sources of the irradiatingunit 1 and the distances among the reflected images detected from theface image.

At step ST14, the eyeglasses feature detecting unit 15 receives the faceimage and the face detection result from the control unit 11, detectspositions of the center points of eyeglasses lenses from within the facedetection frame, and outputs the detection results to the control unit11.

At step ST15, the face orientation detecting unit 17 receives thedetection results of the positions of the respective reflected images,the positions of the center points of the lenses, and the head positionfrom the control unit 11, and first detects the vertical and horizontalface orientation of the passenger. Then, the face orientation detectingunit 17 outputs the detected horizontal angle and vertical angle of theface orientation to the control unit 11. Note that, when the passengerorients his/her face in the direction of the irradiating unit 1, i.e.,when the horizontal angle of the face orientation is 0 degree and thevertical angle of the face orientation is 0 degree, the face orientationdetecting unit 17 outputs the detected horizontal angle, vertical angle,and inclination angle of the face orientation to the control unit 11.

FIGS. 5A, 5B, 5C, 5D, 5E, and 5F are diagrams for explaining apositional relation between a lens of eyeglasses and reflected imagesreflected on the lens. FIG. 6 is a diagram showing positions ofreflected images reflected on eyeglasses lenses when a passenger wearingeyeglasses orients his/her face in various directions.

Here, it is assumed that, when the passenger orients his/her face in thedirection of the irradiating unit 1, i.e., when the horizontal angle andthe vertical angle of the face orientation are 0 degrees, as shown inFIG. 5A, reflected images are located at the center point of theeyeglasses lens. When the passenger orients his/her face leftward withmaintaining its vertical angle to be at 0 degree as viewed from theirradiating unit 1, i.e., when the leftward angle of the faceorientation is 60 degrees, for example, and the vertical angle is 0degree, as shown in FIG. 5B, reflected images are located on the rightside from the center point of the eyeglasses lens. On the other hand,when the passenger orients his/her face rightward, i.e., when therightward angle of the face orientation is 60 degrees, for example, andthe vertical angle is 0 degree, as shown in FIG. 5C, reflected imagesare located on the left side from the center point of the eyeglasseslens.

When the passenger orients his/her face downward with maintaining itshorizontal angle to be at 0 degree as viewed from the irradiating unit1, i.e., when the horizontal angle of the face orientation is 0 degreeand the downward angle is 30 degrees, for example, as shown in FIG. 5D,reflected images are located on the upper side from the center point ofthe eyeglasses lens. On the other hand, when the passenger orientshis/her face upward, i.e., when the horizontal angle of the faceorientation is 0 degree and the upward angle is 30 degrees, for example,as shown in FIG. 5E, reflected images are located on the lower side fromthe center point of the eyeglasses lens.

Further, when the passenger orients his/her face obliquely upward left,i.e., when the leftward angle of the face orientation is 60 degrees andthe upward angle is 30 degrees, for example, as shown in FIG. 5F,reflected images are located on the obliquely downward right side fromthe center point of the eyeglasses lens. Likewise, when the passengerorients his/her face obliquely downward left, the positions of reflectedimages are in a combined state of FIGS. 5B and 5D and are obliquelyupward right side from the center point of the lens, and when thepassenger orients his/her face obliquely upward right, the positions ofreflected images are in a combined state of FIGS. 5C and 5E and areobliquely downward left side from the center point of the lens.

FIG. 7 is a diagram for explaining a method of detecting the horizontalangle and vertical angle of the face orientation, and shows fivereflected images reflected on an eyeglasses lens. x′ is a distance (mm)from the center point of the eyeglasses lens to a reflected image in thehorizontal direction (x-axis direction). y′ is a distance (mm) from thecenter point of the eyeglasses lens to the reflected image in thevertical direction (y-axis direction).

The face orientation detecting unit 17 calculates the horizontal angleθx of the face orientation using equation (1), and calculates thevertical angle θy of the face orientation using equation (2).θx=sin⁻¹(wx×z×xp/R)=sin⁻¹(x′/R)  (1)θy=sin⁻¹(wy×z×yp/R)=sin⁻¹(y′/R)  (2)

Here, wx is a weight used when the units are converted from pixels inthe horizontal direction (x-axis direction) that change in accordancewith the depth direction z to the distance expressed in millimeters, xpis the distance (pixels) from the center point of the eyeglasses lens toa reflected image in the horizontal direction (x-axis direction), wy isa weight used when the units are converted from pixels in the verticaldirection (y-axis direction) that change in accordance with the depthdirection z to the distance expressed in millimeters, yp is the distance(pixels) from the center point of the eyeglasses lens to a reflectedimage in the vertical direction (y-axis direction), z is the distance(mm) from the irradiating unit 1 to the passenger's head in the depthdirection (z-axis direction), and R is the radius (mm) of a typicalhuman head and is a fixed value.

The inclination angle of the face orientation can be detected, only whenthe passenger orients his/her face toward the irradiating unit 1, i.e.,only when the passenger orients his/her face forward.

At step ST16, when the passenger orients his/her face toward theirradiating unit 1, i.e., when the horizontal angle and the verticalangle of the face orientation are 0 degrees (“YES” at step ST16), theface orientation detecting unit 17 proceeds to step ST17 and detects theinclination angle of the face orientation. On the other hand, when thepassenger does not orient his/her face toward the irradiating unit 1,i.e., when the horizontal angle or the vertical angle of the faceorientation is other than 0 degree (“NO” at step ST16), the faceorientation detecting unit 17 ends the process. Note that the faceorientation detecting unit 17 does not need to determine whether thehorizontal angle and vertical angle of the face orientation areprecisely 0 degree, and may determine whether those angles are in apredetermined range including 0 degree.

At step ST17, the face orientation detecting unit 17 detects theinclination angle of the face orientation by comparing the coordinatevalues of the positions in the vertical direction of the respectivereflected images reflected on the eyeglasses lenses, and outputs thedetection result of the inclination angle of the face orientation to thecontrol unit 11.

FIGS. 8A and 8B are views for explaining a method of detecting aninclination angle of the face orientation. FIG. 8A shows a passengerwhose face orientation is as follows: the horizontal angle is 0 degree,the vertical angle is 0 degree, and the inclination angle is 0 degree.FIG. 8B shows a passenger whose face orientation is as follows: thehorizontal angle is 0 degree, the vertical angle is 0 degree, and theleft inclination angle is 30 degrees.

It is assumed that the information about the arrangement of theirradiating unit 1 indicating that, in the five light sources shown inFIG. 2C, three light sources including the center one and both the leftside and right side ones are arranged in a straight line in thehorizontal direction, is provided to the face orientation detecting unit17 in advance.

The face orientation detecting unit 17 determines that the inclinationangle of the face orientation is 0 degree when the coordinate values ofthe positions of respective reflected images of the above-describedthree light sources reflected on the eyeglasses lenses in the verticaldirection are identical as shown in FIG. 8A, i.e., when the respectivereflected images of the above-described three light sources are parallelto the x-axis extending in the horizontal direction. On the other hand,when the coordinate values of the positions of the respective reflectedimages of the above-described three light sources in the verticaldirection are not identical as shown in FIG. 8B, the face orientationdetecting unit 17 calculates the angle formed by the straight line Lthat connects the respective reflected images of the above-describedthree light sources and the x-axis in the horizontal direction, anddetermines the angle as the inclination angle of the face orientation.

By the processes at the above-described steps ST12 to ST17, thepassenger detection device 10 detects the head position and the faceorientation when the passenger wears eyeglasses. Note that the faceorientation detecting unit 17 may only detect the face orientation whichis one of the up, down, left, or right direction, or may also detect theangle in addition to such an orientation.

On the other hand, when the passenger does not wear eyeglasses, thepassenger detection device 10 detects the face orientation of thepassenger in a process at step ST18. At this step ST18, the faceorientation detecting unit 17 receives, through the control unit 11, theface image and the face detection result, detects the face orientationby a known technique, and outputs the detection result to the controlunit 11.

As described above, the passenger detection device 10 according to thefirst embodiment includes: a face image obtaining unit 12 obtaining aface image of a passenger in a vehicle, the face image being captured ina state in which the passenger is irradiated with irradiation light froma plurality of light sources; a reflected-image position detecting unit14 detecting positions of respective reflected images of the pluralityof light sources from the face image obtained by the face imageobtaining unit 12, the reflected images being reflected on eyeglassesworn by the passenger; an eyeglasses feature detecting unit 15detecting, from the face image obtained by the face image obtaining unit12, a position of a center point of a lens as feature information of aneyeglasses; a head position detecting unit 16 detecting a head positionof the passenger on a basis of the positions of respective reflectedimages detected by the reflected-image position detecting unit 14; and aface orientation detecting unit 17 detecting face orientation of thepassenger on a basis of a relation among the head position of thepassenger detected by the head position detecting unit 16, the positionsof respective reflected images detected by the reflected-image positiondetecting unit 14, and the position of the center point of the lensdetected by the eyeglasses feature detecting unit 15. According to thisconfiguration, outside light is not used but irradiation light which isirradiated from a plurality of light sources is used, and thus, the faceorientation of a passenger can always be detected without being affectedby time, weather, and the like. In addition, there is no need to performhigh-precision image processing for extracting vehicle body structuresreflected on eyeglasses lenses like a conventional case, and it isenough for the image processing to detect reflected images that appearas images of white dots on a face image, and thus, a high-performancecamera is not required, and the face orientation detection can beperformed at low cost.

In addition, according to the first embodiment, the irradiating unit 1is configured to have a plurality of light sources arranged in any ofthe vertical, horizontal, and oblique lines. There are the followingfour advantages in arranging the plurality of light sources included inthe irradiating unit 1 in a characteristic shape as described above.

The first advantage is that it becomes easier to distinguish reflectedimages of outside light reflected on eyeglasses lenses and reflectedimages of the irradiating unit 1. The reflected images of outside lightreflected on the eyeglasses lenses almost never form the characteristicshapes such as those shown in FIGS. 2B to 2E and the like. Hence, byarranging the irradiating unit 1 in a characteristic shape, theprobability that reflected images of outside light are erroneouslyidentified as reflected images of the irradiating unit 1 is decreased.

The second advantage is that high-precision image processing becomesunnecessary. In the invention according to Patent Literature 1 describedbefore, for extracting vehicle body structures reflected on lenses byimage processing, it is required to perform high-precision imageprocessing using a high-quality captured image. On the other hand, inthe present embodiment, since reflected images of a characteristic shapesuch as that of the irradiating unit 1 are detected from within a facedetection frame, a camera that captures a high-quality image is notrequired and the image processing time can be reduced.

The third advantage is that even if a monocular camera is used as theimaging unit 2, the distance in the depth direction (z-axis direction)of the head position can be detected. Conventionally, detection of thedistance in the depth direction of the head position requires the use ofa stereo camera or a Time-of-Flight camera that detects a distance onthe basis of the arrival speed of laser light, and it is difficult todetect the distance by a monocular camera. On the other hand, in thepresent embodiment, by arranging a plurality of light sources as theirradiating unit 1, the distance in the depth direction from theirradiating unit 1 to the passenger can be calculated on the basis ofthe distances between reflected images.

The fourth advantage is that the inclination angle of a face orientationwhen the passenger faces forward, i.e., when the passenger orientshis/her face in the direction of the irradiating unit 1, can bedetected. When the irradiating unit 1 is composed of one light source,only the horizontal angle and vertical angle of the face orientation canbe detected. On the other hand, when the irradiating unit 1 is composedof a plurality of light sources arranged in the horizontal direction,the inclination angle of the face orientation can be detected.

Second Embodiment

In the above-described first embodiment, the positions of the centerpoints of lenses are detected as eyeglasses feature information. On theother hand, in this second embodiment, the shapes of lenses or framesare detected as eyeglasses feature information.

FIG. 9 is a block diagram showing an exemplary configuration of apassenger detection system according to the second embodiment of theinvention. In FIG. 9, the same or corresponding portions to those inFIG. 1 are denoted by the same reference signs and description thereofis omitted.

FIG. 10 is a flowchart showing an operation of a passenger detectiondevice 10 according to the second embodiment. Steps ST10 to ST18 in theflowchart shown in FIG. 10 have the same processes as those at stepsST10 to ST18 shown in the flowchart of FIG. 3.

At step ST20, an eyeglasses feature detecting unit 15 a receives,through the control unit 11, a face image and a face detection resultand detects lenses or frames of eyeglasses worn by the passenger fromwithin a face detection frame in the face image.

Then, the eyeglasses feature detecting unit 15 a detects shapes of thelenses or frames of the eyeglasses as feature information indicating theshape of the eyeglasses which corresponds to the face orientation of thepassenger, and outputs the detection result to the control unit 11. Fordetection of shapes of the lenses or frames, the eyeglasses featuredetecting unit 15 a may perform edge detection within the lenses orframes in the face image or may perform a comparison among luminancevalues.

In the following, description is made using the shapes of left and rightlenses as an example of eyeglasses feature information.

FIG. 11 is a diagram for explaining the shapes of left and right lenses.The eyeglasses feature detecting unit 15 a calculates a circumscribedquadrilateral M1 of a right lens of the passenger and detects the lensheights N1 and N2 of the circumscribed quadrilateral M1. Likewise, theeyeglasses feature detecting unit 15 a calculates a circumscribedquadrilateral M2 of a left lens and detects the lens heights N3 and N4of the circumscribed quadrilateral M2. The lens heights N1, N2, N3, andN4 are outputted to the control unit 11 as the shapes of the left andright lenses. Note that the height direction of a lens is the verticaldirection (y-axis direction) in FIG. 2A and the width direction of thelens is the horizontal direction (x-axis direction).

At step ST21, a face orientation detecting unit 17 a determines whetherthe shapes of the left and right lenses when the passenger orientshis/her face toward the irradiating unit 1, i.e., when the horizontalangle of the face orientation is 0 degree and the vertical angle of theface orientation is 0 degree, are stored in a forward-looking datastorage unit 20. The shapes of the left and right lenses for this timeare called forward-looking data.

If the forward-looking data is stored in the forward-looking datastorage unit 20 (“YES” at step ST21), the face orientation detectingunit 17 a proceeds to step ST22, and if the forward-looking data is notstored in the forward-looking data storage unit 20 (“NO” at step ST21),the face orientation detecting unit 17 a proceeds to step ST14.

At step ST22, the face orientation detecting unit 17 a receives, throughthe control unit 11, the detection results of the shapes of the left andright lenses, and compares the shapes of the left and right lenses withthe forward-looking shapes of the left and right lenses stored in theforward-looking data storage unit 20 and thereby detects the horizontalangle and vertical angle of the face orientation of the passenger on thebasis of the change in the shapes of the lenses depending on the faceorientation of the passenger.

FIGS. 12A, 12B, and 12C are diagrams describing a method of detectingface orientation of the present embodiment.

For example, when the passenger orients his/her face to the direction ofthe irradiating unit 1, i.e., when the passenger faces forward in whichthe face orientation is as follows: the horizontal angle is 0 degree andthe vertical angle is 0 degree, as shown in FIG. 12A, the ratio betweenlens heights is (N1:N4)=(1:1).

When the passenger orients his/her face rightward with maintaining itsvertical angle to be at 0 degree, i.e., when the rightward angle of theface orientation is 30 to 60 degrees and the vertical angle of the faceorientation is 0 degree, as shown in FIG. 12B, the ratio between lensheights is (N1:N4)=(10:13), for example, in which N4 is larger than N1.On the other hand, when the passenger orients his/her face leftward,i.e., when the leftward angle of the face orientation is 30 to 60degrees and the vertical angle of the face orientation is 0 degree, theratio between lens heights is (N1:N4)=(13:10), for example, in which N4is smaller than N1.

When the passenger orients his/her face upward with maintaining itshorizontal angle to be at 0 degree, i.e., for example, when thehorizontal angle of the face orientation is 0 degree and the upwardangle of the face orientation is 30 degrees, as shown in FIG. 12C, thelens height N1′ is smaller than the lens height N1 when the passengerfaces forward. In addition, when the passenger orients his/her faceupward, the position at which the height N1′ is detected is an upperposition than a position at which the lens height N1 is detected whenthe passenger faces forward. The vertical angle of the face orientationcan be calculated geometrically using the distance from the irradiatingunit 1 to the passenger and the positions at which the lens heights N1and N1′ are detected, for example.

On the other hand, when the passenger orients his/her face downward withmaintaining its horizontal angle to be at 0 degree, i.e., for example,when the horizontal angle of the face orientation is 0 degree and thedownward angle of the face orientation is 30 degrees, though not shownin the drawings, the lens height N1′ is smaller than the lens height N1when the passenger faces forward. In addition, the position at which theheight N1′ is detected when the passenger orients his/her face downwardis a lower position than the position at which the lens height N1 isdetected when the passenger faces forward.

Note that the ratios between lens heights are only examples and are notlimited to the above-described values.

When the passenger orients his/her face toward the irradiating unit 1,i.e., when the horizontal angle of the face orientation is 0 degree andthe vertical angle of the face orientation is 0 degree (“YES” at stepST16), the face orientation detecting unit 17 a proceeds to step ST17and detects the inclination angle of the face orientation.

At subsequent step ST23, the face orientation detecting unit 17 astores, as forward-looking data, the shapes of the left and right lensesdetected by the eyeglasses feature detecting unit 15 a at step ST20 inthe forward-looking data storage unit 20.

On the other hand, when the passenger does not orient his/her facetoward the irradiating unit 1, i.e., when the horizontal angle of theface orientation is other than 0 degree or the vertical angle of theface orientation is other than 0 degree (“NO” at step ST16), the faceorientation detecting unit 17 a skips steps ST17 and ST23.

As described above, according to the second embodiment, the eyeglassesfeature detecting unit 15 a detects shapes of lenses or frames as thefeature information of the eyeglasses, and the face orientationdetecting unit 17 a detects face orientation of the passenger on a basisof a change in the shapes of lenses or frames which corresponds to faceorientation of the passenger, the shapes of lenses or frames beingdetected by the eyeglasses feature detecting unit 15 a. According tothis configuration, since outside light is not used, the faceorientation of the passenger can always be detected without beingaffected by time, weather, and the like. In addition, since there is noneed to perform high-precision image processing for extracting vehiclebody structures reflected on eyeglasses lenses like a conventional case,a high-quality camera is not required so that the face orientation canbe detected at low cost.

Further, in the case of the configuration of the second embodiment, thedetection accuracy of an angle may be decreased, particularly when thepassenger's face is directed upward or downward, as shown in FIGS. 12Ato 12C. Hence, to improve the detection accuracy of a face orientationangle, the configurations of the first and second embodiments may becombined to each other.

For example, the face orientation detecting unit 17 a performs both offace orientation detection that uses the positions of the center pointsof lenses and face orientation detection that uses the shapes of thelenses or frames. Then, the face orientation detecting unit 17 aoutputs, as a face orientation angle, a weighted average calculated bygiving a weight of “8” for the face orientation angle that uses thepositions of the center points of the lenses, and giving a weight of “2”for the face orientation angle that uses the shapes of the lenses orframes. Note that the weight ratio is not limited to these examples.

In addition, when the reliability of detection of lenses or frames bythe eyeglasses feature detecting unit 15 a is decreased to be lower thana threshold value, the face orientation detecting unit 17 a may reducethe weight of the face orientation angle that uses the shapes of thelenses or frames.

By using not only the positions of the center points of lenses but alsothe shapes of the lenses or frames, the influence due to differences inshape between eyeglasses is reduced and it is possible to correspond toeyeglasses of various shapes. In addition, since both of faceorientation detection that uses the positions of the center points ofthe lenses and face orientation detection that uses the shapes of thelenses or frames are used, the detection accuracy of a face orientationangle is improved compared to the case of using only the shapes of thelenses or frames.

Third Embodiment

In this third embodiment, to improve the detection accuracy of the faceorientation angle, relation between the shapes of lenses or frames andface orientation of a passenger is learned using the positions of thecenter points of eyeglasses lenses.

FIG. 13 is a block diagram showing an exemplary configuration of apassenger detection system according to the third embodiment of theinvention. In FIG. 13, the same or corresponding portions to those inFIG. 1 are denoted by the same reference signs and description thereofis omitted.

FIG. 14 is a flowchart showing an operation of a passenger detectiondevice 10 according to the third embodiment. Steps ST10 to ST18 in theflowchart shown in FIG. 14 are the same processes as those at steps ST10to ST18 shown in the flowchart of FIG. 3.

At step ST30, an eyeglasses feature detecting unit 15 b receives,through the control unit 11, a face image and a face detection resultand detects lenses or frames worn by the passenger from within a facedetection frame in the face image. Then, the eyeglasses featuredetecting unit 15 b detects, as shown in FIG. 11, shapes of the lensesor frames of the eyeglasses as eyeglasses feature information, andoutputs the detection results to the control unit 11.

In the following, description is made using an example case in which theeyeglasses feature information indicates the shapes of left and rightlenses.

At step ST31, a face orientation detecting unit 17 b determines whetherlearning data is stored in a learning result database (DB) 30. Thelearning data will be described later.

If sufficient learning data is stored in the learning result DB 30(“YES” at step ST31), the face orientation detecting unit 17 b proceedsto step ST32, and if sufficient learning data is not stored in thelearning result DB 30 (“NO” at step ST31), the face orientationdetecting unit 17 b proceeds to step ST14. The sufficient learning dataas mentioned herein refers to data which allows to interpolate thehorizontal angle and vertical angle of the face orientationcorresponding to the shapes of the left and right lenses that are notstored as learning data at step ST32 described later.

At step ST33, the face orientation detecting unit 17 b generateslearning data, in which the horizontal angle and vertical angle of theface orientation detected at step ST15 and the shapes of the left andright lenses detected at step ST30 are associated with one another, andstores the learning data in the learning result DB 30.

At step ST32, the face orientation detecting unit 17 b detects thehorizontal angle and vertical angle of the face orientationcorresponding to the shapes of the left and right lenses detected atstep ST30, using the learning data stored in the learning result DB 30,and outputs the detection results to the control unit 11. At this time,the face orientation detecting unit 17 b may interpolate the horizontalangle and vertical angle of the face orientation corresponding to theshapes of the left and right lenses that are not stored as learningdata, using the horizontal angle and vertical angle of the faceorientation corresponding to the shapes of the left and right lensesthat are stored as learning data.

By the above, according to the third embodiment, the eyeglasses featuredetecting unit 15 b detects positions of center points of lenses andshapes of the lenses or frames as the feature information of theeyeglasses, and the face orientation detecting unit 17 b detects faceorientation of the passenger on a basis of relation among the headposition of the passenger detected by the head position detecting unit16, the positions of respective reflected images detected by thereflected-image position detecting unit 14, and the positions of centerpoints of lenses detected by the eyeglasses feature detecting unit 15 b,and performs learning, using the detected face orientation, relationbetween the shapes of the lenses or frames detected by the eyeglassesfeature detecting unit 15 b and the face orientation of the passenger.

Then, the face orientation detecting unit 17 b is configured to detect,using the learned data, face orientation corresponding to the shapes ofthe lenses or frames detected by the eyeglasses feature detecting unit15 b. According to this configuration, since outside light is not used,the face orientation of the passenger can always be detected withoutbeing affected by time, weather, and the like. In addition, since thereis no need to perform high-precision image processing for extractingvehicle body structures reflected on eyeglasses lenses like theconventional case, a high-quality camera is not required, so that theface orientation can be detected at low cost. Furthermore, sincerelation between the shapes of lenses or frames and the face orientationis learned, the detection accuracy of the face orientation angle isimproved compared to the second embodiment.

Fourth Embodiment

In this fourth embodiment, the shapes of temples placed over ears aredetected as eyeglasses feature information.

FIG. 15 is a block diagram showing an exemplary configuration of apassenger detection system according to the fourth embodiment of theinvention. In FIG. 15, the same or corresponding portions to those inFIG. 1 are denoted by the same reference signs and description thereofis omitted.

FIGS. 16 and 17 are flowcharts showing an operation of a passengerdetection device 10 according to the fourth embodiment. Steps ST10 toST18 in the flowcharts shown in FIGS. 16 and 17 have the same processesas those at steps ST10 to ST18 shown in the flowchart of FIG. 3.

As shown in FIG. 6, when a passenger orients his/her face rightward orleftward by a large angle as viewed from the side where the irradiatingunit 1 and the imaging unit 2 are installed, the length of an eyeglassestemple increases, and when the passenger orients his/her face forward,the length of the eyeglasses temple decrease.

In addition, when the passenger orients his/her face upward or downwardby a large angle, the orientation of the eyeglasses temples becomesupward or downward, and when the passenger orients his/her face forward,the orientation of the eyeglasses temples becomes horizontal.

Thus, the horizontal angle of the face orientation of the passenger canbe detected on the basis of the lengths of the eyeglasses temples, andthe vertical angle of the face orientation of the passenger can bedetected on the basis of the orientation of the eyeglasses temples.

At step ST40, an eyeglasses feature detecting unit 15 c receives,through the control unit 11, a face image and a face detection resultand detects lenses or frames worn by the passenger from within the facedetection frame in the face image. Then, the eyeglasses featuredetecting unit 15 b detects, as shown in FIG. 11, shapes of the lensesor frames as feature information indicating the shape of eyeglasseswhich corresponds to the face orientation of the passenger, and outputsthe feature information to the control unit 11.

In the following, an example in which eyeglasses feature informationindicates the lens heights and lens widths which show the shapes of theleft and right lenses is described. The lens heights are four sides (N1,N2, N3, and N4) extending in the vertical direction of the circumscribedquadrilaterals M1 and M2 in FIG. 11, and the lens widths are four sidesextending in the horizontal direction of the circumscribedquadrilaterals M1 and M2.

If the eyeglasses feature detecting unit 15 c can detect the lensheights and lens widths of the left and right lenses (“YES” at stepST40), the eyeglasses feature detecting unit 15 c proceeds to step ST41,and if the eyeglasses feature detecting unit 15 c cannot detect any ofthe lens heights and lens widths (“NO” at step ST40), the eyeglassesfeature detecting unit 15 c proceeds to step ST46. When, for example,the passenger orients his/her face leftward or rightward by a largeangle, any of the lens heights and lens widths of the left and rightlenses may not be able to be detected. If any of the lens heights andlens widths of the left and right lenses cannot be detected, then thepositions of the center points of the lenses cannot also be detected. Inthe present embodiment, even in such a case, by steps ST46 to ST48 whichwill be described later, the face orientation can be detected usingdetection results of eyeglasses temples.

At step ST41, the eyeglasses feature detecting unit 15 c detects thelengths and orientation of the temples of the eyeglasses worn by thepassenger from within the face detection frame in the face image, andoutputs the detection results as eyeglasses feature information to thecontrol unit 11. To detect the eyeglasses temples, the eyeglassesfeature detecting unit 15 c may perform edge detection in the vicinityof the lenses or frames in the face image or may perform a comparisonamong luminance values.

At step ST42, a face orientation detecting unit 17 c determines whetherforward-looking data is stored in the forward-looking data storage unit40. The forward-looking data shows the shapes of the left and rightlenses when the passenger orients his/her face toward the irradiatingunit 1, i.e., when the horizontal angle of the face orientation is 0degree and the vertical angle of the face orientation is 0 degree.

If the forward-looking data is stored in the forward-looking datastorage unit 40 (“YES” at step ST42), the face orientation detectingunit 17 c proceeds to step ST43, and if the forward-looking data is notstored in the forward-looking data storage unit 40 (“NO” at step ST42),the face orientation detecting unit 17 c proceeds to step ST14.

At step ST43, the face orientation detecting unit 17 c receives, throughthe control unit 11, the detection results of the shapes of the left andright lenses, and compares the shapes of the left and right lenses withthe forward-looking shapes of the left and right lenses stored in theforward-looking data storage unit 40 and thereby detects the horizontalangle and vertical angle of the face orientation of the passenger on thebasis of the change in the shapes depending on the face orientation ofthe passenger.

At step ST44, the face orientation detecting unit 17 c generateslearning data in which the horizontal angle and vertical angle of theface orientation detected at step ST15 or step ST43 and the detectionresults of the eyeglasses temples detected at step ST41 are associatedwith one another, and stores the learning data in a learning result DB41.

At step ST45, the face orientation detecting unit 17 c stores the shapesof the left and right lenses when the passenger orients his/her facetoward the side of the irradiating unit 1, i.e., when the horizontalangle of the face orientation is 0 degree and the vertical angle of theface orientation is 0 degree, in the forward-looking data storage unit40 as forward-looking data.

At step ST46, the face orientation detecting unit 17 c determineswhether learning data is stored in the learning result DB 41. Ifsufficient learning data is stored in the learning result DB 41 (“YES”at step ST46), the face orientation detecting unit 17 c proceeds to stepST47, and if sufficient learning data is not stored in the learningresult DB 41 (“NO” at step ST46), the face orientation detecting unit 17c ends the process.

At step ST47, the face orientation detecting unit 17 c detectseyeglasses temples similarly to the process in step ST41.

At step ST48, the face orientation detecting unit 17 c detects, usingthe learning data stored in the learning result DB 41, the horizontalangle and vertical angle of the face orientation corresponding to thelengths and orientation of the eyeglasses temples detected at step ST47,and outputs the detection results to the control unit 11. At this time,the face orientation detecting unit 17 c may interpolate the horizontalangle and vertical angle of the face orientation corresponding to thelengths and orientations of the eyeglasses temples that are not storedas learning data, using the horizontal angle and vertical angle of theface orientation corresponding to the lengths and orientations of theeyeglasses temples that are stored as learning data.

Note that although in the example of FIGS. 16 and 17 the faceorientation detecting unit 17 c learns relation between the shapes ofeyeglasses temples and face orientation using both of the shapes oflenses or frames and the positions of the center points of the lenses,the face orientation detecting unit 17 c may perform learning using onlyone of them.

By the above, according to the fourth embodiment, the eyeglasses featuredetecting unit 15 c detects, as the feature information of theeyeglasses, shapes of temples placed over ears, and at least either oneof shapes of lenses or frames and positions of center points of thelenses. The face orientation detecting unit 17 c detects faceorientation of the passenger on a basis of the shapes of lenses orframes detected by the eyeglasses feature detecting unit 15 c, ordetects face orientation of the passenger on a basis of relation amongthe head position of the passenger detected by the head positiondetecting unit 16, the positions of respective reflected images detectedby the reflected-image position detecting unit 14, and the positions ofcenter points of the lenses detected by the eyeglasses feature detectingunit 15 c, and performs learning, using the detected face orientation ofthe passenger, relation between the shapes of temples detected by theeyeglasses feature detecting unit 15 c and the face orientation. Theface orientation detecting unit 17 c detects, using the data obtained bythe learning, the face orientation corresponding to the shapes oftemples detected by the eyeglasses feature detecting unit 15 c. By thisconfiguration, even when a part of the shapes of lenses or frames or thepositions of respective reflected images cannot be detected, faceorientation detection that uses the shapes of the temples can beperformed on the basis of learning results. Therefore, a large faceorientation angle that is difficult to detect with high accuracy in theabove-described first and second embodiments can also be accuratelydetected.

Finally, with reference to FIG. 18, an exemplary hardware configurationof the passenger detection system according to the respectiveembodiments of the invention will be described.

The irradiating unit 1 in the passenger detection system is anilluminating device 101 including a plurality of light sources. Theimaging unit 2 in the passenger detection system is a camera 102. Theface image obtaining unit 12 in the passenger detection device 10 is aninput device 103 that is connected to the camera 102 and that obtains aface image from the camera 102 for inputting the face image to thepassenger detection device 10. The control unit 11, the face detectingunit 13, the reflected-image position detecting unit 14, the eyeglassesfeature detecting unit 15, 15 a to 15 c, the head position detectingunit 16, and the face orientation detecting unit 17, 17 a to 17 c in thepassenger detection device 10 are a processor 104 that executes programsstored in a memory 105. The forward-looking data storage unit 20, 40 andthe learning result DB 30, 41 in the passenger detection device 10 arethe memory 105.

The functions of the control unit 11, the face detecting unit 13, thereflected-image position detecting unit 14, the eyeglasses featuredetecting unit 15, 15 a to 15 c, the head position detecting unit 16,and the face orientation detecting unit 17, 17 a to 17 c are implementedby software, firmware, or a combination of software and firmware. Thesoftware or firmware is described as a program and stored in the memory105. The processor 104 implements the function of each unit by readingout and executing a program stored in the memory 105. Namely, thepassenger detection device 10 includes the memory 105 for storingprograms by which each step shown in FIG. 3, 10, 14, 16, or 17 isconsequently performed when the programs are executed by the processor104. It can also be said that the programs causes a computer to performprocedures or methods performed by the respective units of the passengerdetection device 10.

The processor 104 is also called a central processing unit (CPU), aprocessing device, an arithmetic device, a microprocessor, amicrocomputer, a digital signal processor (DSP), or the like. The memory105 may be, for example, a nonvolatile or volatile semiconductor memorysuch as a random access memory (RAM), a read only memory (ROM), a flashmemory, an erasable programmable ROM (EPROM), or an electrically EPROM(EEPROM), or may be a magnetic disc such as a hard disc or a flexibledisc, or an optical disc such as a MiniDisc, a compact disc (CD), or adigital versatile disc (DVD).

Note that, in the present invention, a free combination of theembodiments, modifications of any component of the embodiments, oromissions of any component of the embodiments are possible within thescope of the invention.

INDUSTRIAL APPLICABILITY

A passenger detection device according to the invention detects a faceorientation of a passenger in a vehicle using a face image which iscaptured in a state in which the passenger is irradiated withirradiation light from a plurality of light sources, and thus, issuitable for use as, for example, a passenger detection device forvehicles such as automobiles, railway vehicles, ships, or aircrafts.

REFERENCE SIGNS LIST

1: Irradiating unit, 2: Imaging unit, 10: Passenger detection device,11: Control unit, 12: Face image obtaining unit, 13: Face detectingunit, 14: Reflected-image position detecting unit, 15 and 15 a to 15 c:Eyeglasses feature detecting unit, 16: Head position detecting unit, 17and 17 a to 17 c: Face orientation detecting unit, 20 and 40:Forward-looking data storage unit, 30 and 41: Learning result DB, 101:Illuminating device, 102: Camera, 103: Input device, 104: Processor, and105: Memory

The invention claimed is:
 1. A passenger detection device comprising: aface image obtaining unit obtaining a face image of a passenger in avehicle, the face image being captured in a state in which the passengeris irradiated with irradiation light from a plurality of light sources;a reflected-image position detecting unit detecting positions ofrespective reflected images of the plurality of light sources from theface image obtained by the face image obtaining unit, the reflectedimages being reflected on eyeglasses worn by the passenger; aneyeglasses feature detecting unit detecting, from the face imageobtained by the face image obtaining unit, feature informationindicating a position or shape of the eyeglasses which corresponds toface orientation of the passenger; a head position detecting unitdetecting a head position of the passenger on a basis of the positionsof respective reflected images detected by the reflected-image positiondetecting unit; and a face orientation detecting unit detecting faceorientation of the passenger on a basis of the head position of thepassenger detected by the head position detecting unit, the positions ofrespective reflected images detected by the reflected-image positiondetecting unit, and the feature information of the eyeglasses detectedby the eyeglasses feature detecting unit.
 2. The passenger detectiondevice according to claim 1, wherein the eyeglasses feature detectingunit detects positions of center points of lenses as the featureinformation of the eyeglasses, and the face orientation detecting unitdetects face orientation of the passenger on a basis of relation amongthe head position of the passenger detected by the head positiondetecting unit, the positions of respective reflected images detected bythe reflected-image position detecting unit, and the positions of centerpoints of lenses detected by the eyeglasses feature detecting unit. 3.The passenger detection device according to claim 1, wherein theeyeglasses feature detecting unit detects shapes of lenses or frames asthe feature information of the eyeglasses, and the face orientationdetecting unit detects face orientation of the passenger on a basis of achange in the shapes of lenses or frames which corresponds to faceorientation of the passenger, the shapes of lenses or frames beingdetected by the eyeglasses feature detecting unit.
 4. The passengerdetection device according to claim 1, wherein the eyeglasses featuredetecting unit detects positions of center points of lenses and shapesof the lenses or frames as the feature information of the eyeglasses,and the face orientation detecting unit detects face orientation of thepassenger on a basis of relation among the head position of thepassenger detected by the head position detecting unit, the positions ofrespective reflected images detected by the reflected-image positiondetecting unit, and the positions of center points of lenses detected bythe eyeglasses feature detecting unit, and performs learning, using thedetected face orientation of the passenger, relation between the shapesof the lenses or frames detected by the eyeglasses feature detectingunit and the face orientation.
 5. The passenger detection deviceaccording to claim 4, wherein the face orientation detecting unitdetects, using the data obtained by the learning, the face orientationcorresponding to the shapes of the lenses or frames detected by theeyeglasses feature detecting unit.
 6. The passenger detection deviceaccording to claim 1, wherein the eyeglasses feature detecting unitdetects, as the feature information of the eyeglasses, shapes of templesplaced over ears, and at least either one of shapes of lenses or framesand positions of center points of the lenses, and the face orientationdetecting unit detects face orientation of the passenger on a basis ofthe shapes of lenses or frames detected by the eyeglasses featuredetecting unit, or detects face orientation of the passenger on a basisof relation among the head position of the passenger detected by thehead position detecting unit, the positions of respective reflectedimages detected by the reflected-image position detecting unit, and thepositions of center points of the lenses detected by the eyeglassesfeature detecting unit, and performs learning, using the detected faceorientation of the passenger, relation between the shapes of templesdetected by the eyeglasses feature detecting unit and the faceorientation.
 7. The passenger detection device according to claim 6,wherein the face orientation detecting unit detects, using the dataobtained by the learning, the face orientation corresponding to theshapes of temples detected by the eyeglasses feature detecting unit. 8.A passenger detection system comprising: an irradiating unit irradiatinga passenger in a vehicle with irradiation light from a plurality oflight sources; an imaging unit capturing an image of the passenger beingirradiated with the irradiation light from the irradiating unit; a faceimage obtaining unit obtaining a face image of the passenger captured bythe imaging unit; a reflected-image position detecting unit detectingpositions of respective reflected images of the plurality of lightsources from the face image obtained by the face image obtaining unit,the reflected images being reflected on eyeglasses worn by thepassenger; an eyeglasses feature detecting unit detecting, from the faceimage obtained by the face image obtaining unit, feature informationindicating a position or shape of the eyeglasses which corresponds toface orientation of the passenger; a head position detecting unitdetecting a head position of the passenger on a basis of the positionsof respective reflected images detected by the reflected-image positiondetecting unit; and a face orientation detecting unit detecting faceorientation of the passenger on a basis of the head position of thepassenger detected by the head position detecting unit, the positions ofrespective reflected images detected by the reflected-image positiondetecting unit, and the feature information of the eyeglasses detectedby the eyeglasses feature detecting unit.
 9. The passenger detectionsystem according to claim 8, wherein the irradiating unit has aplurality of light sources arranged to form any of vertical, horizontal,and oblique lines.
 10. A passenger detection method comprising the stepsof: obtaining, by a face image obtaining unit, a face image of apassenger in a vehicle, the face image being captured in a state inwhich the passenger is irradiated with irradiation light from aplurality of light sources; detecting, by a reflected-image positiondetecting unit, positions of respective reflected images of theplurality of light sources from the face image obtained by the faceimage obtaining unit, the reflected images being reflected on eyeglassesworn by the passenger; detecting, by an eyeglasses feature detectingunit, feature information indicating a position or shape of theeyeglasses which corresponds to face orientation of the passenger, fromthe face image obtained by the face image obtaining unit; detecting, bya head position detecting unit, a head position of the passenger on abasis of the positions of respective reflected images detected by thereflected-image position detecting unit; and detecting, by a faceorientation detecting unit, face orientation of the passenger on a basisof the head position of the passenger detected by the head positiondetecting unit, the positions of respective reflected images detected bythe reflected-image position detecting unit, and the feature informationof the eyeglasses detected by the eyeglasses feature detecting unit.